Pen-based systems allow for fluid and expressive input, making them effective tools for many tasks, such as note taking and design sketching. In inking applications it is easy and intuitive to create ink strokes, since it is similar to physically writing on paper. However, the inherent differences of pen-based interfaces add complications to the task of ink selection which need to be addressed.
The first difficulty is in informing the system that actions are not to be considered ink per se, but are intended to define a selection. Without supplementary buttons, users are often required to make roundtrips to bordering tool bars to transition between these modes, which could have undesirable costs. Furthermore, mode errors inevitably occur when the user forgets to activate the selection mode, activates it by accident (e.g., by mistakenly pressing the pens barrel button while writing), or forgets to turn off “selection mode” when done selecting objects.
The other difficulty is specifying the desired selection to the system. Document objects, such as diagrams and words, can consist of multiple individual ink strokes, potentially resulting in unexpected results when the user tries to select them. For example, a user may try to select and move a word, only to find that one stroke of a character, such as a dot above an ‘i’ was left behind. Furthermore, words and illustrations may be densely clustered together or even overlap. Selecting a specific set of ink strokes within such data with a traditional lasso tool, if possible at all, can require a constrained “steering task”, which can be time-consuming and error-prone.
Numerous techniques have been developed for selection in pen-based interfaces. In early systems, the scope of a command was defined by encircling desired objects to specify a “collective scope”. This type of freeform lasso selection has been commonly employed, but a difficulty with such selection techniques is that in a dense area of ink stokes, for example, users may need to follow a constrained path to ensure that only their ink strokes are selected. Accot's steering law (e.g., a predictive model of how quickly one can navigate, or steer, through a two-dimensional tunnel/maze) would suggest that the performance time of this task can be constrained by the width of the “tunnel”. Accordingly, the narrower the tunnel, the greater the performance time required to complete the task.
While lassoing may be the most common approach, alternatives have also be designed and studied. Tapping requires users to tap down on a desired selection. Tapping has been shown to perform well in comparison to lasso selection when selecting discrete and non-cohesive targets that cannot be circled with a single lasso. However, this approach is not so appropriate for objects consisting of multiple ink strokes, as in such cases a system must typically interpret the desired scope of the user's selection.
Crossing is another alternative that is similar to lassoing but instead of circling the desired selection, the user crosses it. This technique has been explored in pen-based interfaces; however for selection of ink, it can be difficult to cross each atomic element of a compound selection with a single stroke.
An impediment associated with the above selection techniques is the continuous nature of ink data. A document containing freeform ink lacks delimiters which typically defines characters, words and diagrams, and so a system's knowledge of which strokes are related and how remains uncertain. This can cause unexpected results when the user tries to select from this input. For instance, a user may attempt to select and move a word, only to find that one stroke of a character is left behind or orphaned.
One typical solution to this difficulty has been to use grouping—where multiple objects are combined to form explicit and system recognized higher-level structures. In such applications, the grouping can be done implicitly by the system—the system attempts to interpret the data and forms groups of strokes, such as words, sentences, or diagrams. A potential problem with such systems arises when objects are not grouped correctly. For example, a stroke may get mistakenly grouped and selected with a word when it actually belongs to a nearby diagram. Furthermore, once groups are formed, it can be difficult for the user to select elements from the group, such as, for instance, the first character of a word.
Another difficulty with traditional selection techniques has been that ink data commonly contain overlapping strokes, such as text appearing over diagrams. In many cases this can make selections difficult or even impossible. One approach to counter this deficiency has been to employ a tumble and splatter technique spread over overlapping objects to allow users access to desired objects. However, these techniques do not address pen-based interfaces specifically, and moreover, require the user to enter an explicit mode using a button.
Generally, the user must inform the system that their actions with the pointing device (e.g., mouse, stylus, pen, and the like) should be interpreted as either ink or selection, before the associated stroke occurs. The traditional technique for doing so is to travel to a bordering toolbar to click on an icon. Unfortunately, this is time-consuming, physically demanding, and can distract the user's attention from their primary task. Nevertheless, a number of alternatives are available, such as, for instance, using a barrel button, holding down a stylus or pen for a dwell timeout period, pressing a supplementary button with a non-dominant hand, using pressure, and using the eraser of a pen. Comparative investigations have indicated that pressing a button with a non-dominant hand offers the fastest performance, and best trade-off with errors.
Button-free alternatives are also available. Gestural user interfaces typically interpret implicitly the intent from a user's actions, so an explicit mode transition is not generally required. Research has indicated that users have a general preference for such “inferred mode protocols”. However, further research has also suggested that users also preferred an explicit mode change over an inferred mode protocol by a margin of two to one. A challenge associated with gestural interfaces is that misinterpretations can cause unexpected results. Furthermore, such interfaces tend to introduce hidden states and specific orderings of operations that can be counterintuitive or unclear to the user.
A tracking menu is an example of a non-gestural approach that can give users localized access to commands and modes, but it is a mode itself that must be explicitly enabled, and cannot be fluidly integrated with ink input. Hover widgets allow users to access system functionality such as mode switches through localized gestures. The gestures can be carried out in the tracking state of the device, distinguishing them from ink input. Regardless of the technique utilized to transition between modes, if the user forgets or misclassifies the current state of a system, modes can result in errors—errors that can occur even with kinesthetically maintained modes.